Parsers in TEX and using CWEB for general pretty-printing

The need to process formally structured languages inside TEX documents is neither new nor uncommon. Several graphics extensions for TEX (and LTEX) have introduced a variety of small specialized languages for their purposes that depend on simple (and not so simple) interpreters coded as TEX macros. A number of pretty-printing macros take advantage of different parsing techniques to achieve their goals (see [Go], [Do], and [Wo]). Efforts to create general and robust parsing frameworks inside TEX go back to the origins of TEX itself. A well-known BASIC subset interpreter, BASIX (see [Gr]) was written as a demonstration of the flexibility of TEX as a programming language and a showcase of TEX’s ability to handle a variety of abstract data structures. On the other hand, a relatively recent addition to the LTEX toolbox, l3regex (see [La]), provides a powerful and very general way to perform regular expression matching in LTEX, which can be used (among other things) to design parsers and scanners. Paper [Go] contains a very good overview of several approaches to parsing and tokenizing in TEX and outlines a universal framework for parser design using TEX macros. In an earlier article (see [Wo]), Marcin Woliński describes a parser creation suite paralleling the technique used by CWEB (CWEB’s ‘grammar’ is hard-coded into CWEAVE, whereas Woliński’s approach is more general). One commonality between these two methods is a highly customized tokenizer (or scanner) used as the input to the parser proper. Woliński’s design uses a finite automaton as the scanner engine with a ‘manually’ designed set of states. No backing up mechanism was provided, so matching, say, the longest input would require some custom coding (it is, perhaps, worth mentioning here that a backup mechanism is all one needs to turn any regular language scanner into a general CWEB-type parser). The scanner in [Go] was designed mainly with efficiency in mind TUGboat, Volume 35 (2014), No. 1 71